Output of a neural network method for deep odometry assisted by static scene optical flow

ABSTRACT

A method of visual odometry for a non-transitory computer readable storage medium storing one or more programs is disclosed. The one or more programs includes instructions, which when executed by a computing device, causes the computing device to perform the following steps comprising: performing data alignment among sensors including a LiDAR, cameras and an IMU-GPS module; collecting image data and generating point clouds; processing, in the IMU-GPS module, a pair of consecutive images in the image data to recognize pixels corresponding to a same point in the point clouds; and establishing an optical flow for visual odometry.

PRIORITY/RELATED DOCUMENTS

This patent application incorporates by reference in their entiretiesand claims priority to these co-pending patent applications filed onSep. 13, 2017, including the following: (1) “Data Acquisition and Inputof Neural Network Method for Deep Odometry Assisted by Static SceneOptical Flow;” (2) “Data Acquisition and Input of Neural Network Systemfor Deep Odometry Assisted by Static Scene Optical Flow;” (3) “NeuralNetwork Architecture Method for Deep Odometry Assisted by Static SceneOptical Flow;” (4) “Neural Network Architecture System for Deep OdometryAssisted by Static Scene Optical Flow;” (5) “Output of a Neural NetworkSystem for Deep Odometry Assisted by Static Scene Optical Flow;” (6)“Training and Testing of a Neural Network Method for Deep OdometryAssisted by Static Scene Optical Flow;” and (7) “Training and Testing ofa Neural Network System for Deep Odometry Assisted by Static SceneOptical Flow,” and all with the same inventor(s).

FIELD OF THE DISCLOSURE

The field of the disclosure is in general related to autonomous vehiclesand, in particular, to a method and system for deep odometry assisted bystatic scene optical flow.

BACKGROUND OF THE DISCLOSURE

In recent years, an increasing amount of interest and research efforthas been put toward intelligent or autonomous vehicles. With thecontinuous progress in autonomous technology, robot sensors aregenerating increasing amounts of real-world data. Autonomous vehicleresearch is highly dependent on the vast quantities of real-world datafor development, testing and validation of algorithms before deploymenton public roads. However, the cost of processing and analyzing thesedata, including developing and maintaining a suitable autonomous vehicleplatform, regular calibration and data collection procedures, andstoring the collected data, is so high that few research groups canmanage it. Following the benchmark-driven approach of the computervision community, a number of vision-based autonomous driving datasetshave been released. Some existing datasets, however, may not be wellgeneralized to different environments. Moreover, hand-crafted featuresmay be employed to extract keypoints and descriptors, and find matchingpoints to solve motion parameters. Such feature-based methods fail whena scene has no salient keypoints.

BRIEF SUMMARY OF THE DISCLOSURE

Various objects, features, aspects and advantages of the presentembodiment will become more apparent from the following detaileddescription of embodiments of the embodiment, along with theaccompanying drawings in which like numerals represent like components.

Embodiments of the present disclosure provide a method of visualodometry for a non-transitory computer readable storage medium storingone or more programs. The one or more programs includes instructions,which when executed by a computing device, causes the computing deviceto perform the following steps comprising: performing data alignmentamong sensors including a LiDAR, cameras and an IMU-GPS module;collecting image data and generating point clouds; processing a pair ofconsecutive images in the image data to recognize pixels correspondingto a same point in the point clouds; and establishing an optical flowfor visual odometry.

In an embodiment, the method further includes: receiving a first imageof a first pair of image frames, and extracting representative featuresfrom the first image of the first pair in a first convolution neuralnetwork (CNN); and receiving a second image of the first pair, andextracting representative features from the second image of the firstpair in the first CNN.

In another embodiment, the method further includes: merging, in thefirst merge module, outputs from the first CNN; and decreasing featuremap size in a second CNN.

In yet another embodiment, the method further includes: generating afirst flow output for each layer in a first deconvolution neural network(DNN).

In still another embodiment, the method further includes: merging, in asecond merge module, outputs from the second CNN and the first DNN, andgenerating a first motion estimate.

In yet still another embodiment, the method further includes: generatinga second flow output for each layer in a second DNN, the second flowoutput serves as a first optical flow prediction.

In still yet another embodiment, the method further includes: inresponse to the first motion estimate, generating a first set of motionparameters associated with the first pair in a recurrent neural network(RNN).

In a further embodiment, the method further includes: training thevisual odometry model by using at least one of the first optical flowprediction and the first set of motion parameters.

In an embodiment, the method further includes: receiving a first imageof a second pair of image frames, and extracting representative featuresfrom the first image of the second pair in the first CNN; and receivinga second image of the second pair, and extracting representativefeatures from the second image of the second pair in the first CNN.

In another embodiment, the method further includes: merging, in thefirst merge module, outputs from the first CNN; and decreasing featuremap size in the second CNN.

In yet another embodiment, the method further includes: generating afirst flow output for each layer in the first DNN.

In still another embodiment, the method further includes: merging, inthe second merge module, outputs from the second CNN and the first DNN,and generating a second motion estimate.

In yet still another embodiment, the method further includes: generatinga second flow output for each layer in the second DNN, the second flowoutput serves as a second optical flow prediction.

In still yet another embodiment, the method further includes: inresponse to the second motion estimate and the first set of motionparameters, generating a second set of motion parameters associated withthe second pair in the RNN.

In a further embodiment, the method further includes: training thevisual odometry model by using at least one of the second optical flowprediction and the second set of motion parameters.

BRIEF DESCRIPTION OF THE DRAWINGS

It should be noted that the drawing figures may be in simplified formand might not be to precise scale. In reference to the disclosureherein, for purposes of convenience and clarity only, directional termssuch as top, bottom, left, right, up, down, over, above, below, beneath,rear, front, distal, and proximal are used with respect to theaccompanying drawings. Such directional terms should not be construed tolimit the scope of the embodiment in any manner.

FIG. 1 is a flow diagram showing a method of visual odometry, inaccordance with an embodiment;

FIG. 2 is a block diagram of a system for visual odometry, in accordancewith an embodiment;

FIG. 3A is a block diagram showing the system illustrated in FIG. 2 inmore detail;

FIG. 3B is a schematic block diagram showing operation of the systemillustrated in FIG. 3A;

FIG. 4 is a flow diagram showing a method for visual odometry, inaccordance with still another embodiment;

FIG. 5 is a flow diagram showing a method for visual odometry, inaccordance with yet another embodiment;

FIG. 6 is a flow diagram showing a method of visual odometry, inaccordance with yet still another embodiment;

FIG. 7 is a flow diagram showing a method of visual odometry, inaccordance with a further embodiment;

FIGS. 8A and 8B are flow diagrams showing a method of visual odometry,in accordance with a still further embodiment; and

FIG. 9 is a block diagram of a system for generating a ground truthdataset for motion planning, in accordance with some embodiments.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The embodiment and its various embodiments can now be better understoodby turning to the following detailed description of the embodiments,which are presented as illustrated examples of the embodiment defined inthe claims. It is expressly understood that the embodiment as defined bythe claims may be broader than the illustrated embodiments describedbelow.

Any alterations and modifications in the described embodiments, and anyfurther applications of principles described in this document arecontemplated as would normally occur to one of ordinary skill in the artto which the disclosure relates. Specific examples of components andarrangements are described below to simplify the present disclosure.These are, of course, merely examples and are not intended to belimiting. For example, when an element is referred to as being“connected to” or “coupled to” another element, it may be directlyconnected to or coupled to the other element, or intervening elementsmay be present.

In the drawings, the shape and thickness may be exaggerated for clarityand convenience. This description will be directed in particular toelements forming part of, or cooperating more directly with, anapparatus in accordance with the present disclosure. It is to beunderstood that elements not specifically shown or described may takevarious forms. Reference throughout this specification to “oneembodiment” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment.

In the drawings, the figures are not necessarily drawn to scale, and insome instances the drawings have been exaggerated and/or simplified inplaces for illustrative purposes. One of ordinary skill in the art willappreciate the many possible applications and variations of the presentdisclosure based on the following illustrative embodiments of thepresent disclosure.

The appearances of the phrases “in one embodiment” or “in an embodiment”in various places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments. It should be appreciated that the followingfigures are not drawn to scale; rather, these figures are merelyintended for illustration.

It will be understood that singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. Furthermore, relative terms, such as “bottom” and“top,” may be used herein to describe one element's relationship toother elements as illustrated in the Figures.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure belongs. It willbe further understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and thepresent disclosure, and will not be interpreted in an idealized oroverly formal sense unless expressly so defined herein.

Many alterations and modifications may be made by those having ordinaryskill in the art without departing from the spirit and scope of theembodiment. Therefore, it must be understood that the illustratedembodiment has been set forth only for the purposes of example and thatit should not be taken as limiting the embodiment as defined by thefollowing claims. For example, notwithstanding the fact that theelements of a claim are set forth below in a certain combination, itmust be expressly understood that the embodiment includes othercombinations of fewer, more, or different elements, which are disclosedherein even when not initially claimed in such combinations.

The words used in this specification to describe the embodiment and itsvarious embodiments are to be understood not only in the sense of theircommonly defined meanings, but to include by special definition in thisspecification structure, material or acts beyond the scope of thecommonly defined meanings. Thus if an element can be understood in thecontext of this specification as including more than one meaning, thenits use in a claim must be understood as being generic to all possiblemeanings supported by the specification and by the word itself.

The definitions of the words or elements of the following claimstherefore include not only the combination of elements which areliterally set forth, but all equivalent structure, material or acts forperforming substantially the same function in substantially the same wayto obtain substantially the same result.

In this sense it is therefore contemplated that an equivalentsubstitution of two or more elements may be made for any one of theelements in the claims below or that a single element may be substitutedfor two or more elements in a claim. Although elements may be describedabove as acting in certain combinations and even initially claimed assuch, it is to be expressly understood that one or more elements from aclaimed combination can in some cases be excised from the combinationand that the claimed combination may be directed to a subcombination orvariation of a subcombination.

Reference is now made to the drawings wherein like numerals refer tolike parts throughout.

As used herein, the term “wireless” refers to wireless communication toa device or between multiple devices. Wireless devices may be anchoredto a location and/or hardwired to a power system, depending on the needsof the business, venue, event or museum. In one embodiment, wirelessdevices may be enabled to connect to Internet, but do not need totransfer data to and from Internet in order to communicate within thewireless information communication and delivery system.

As used herein, the term “Smart Phone” or “smart phone” or “mobiledevice(s)” or “cellular phone” or “cellular” or “mobile phone” or thelike refers to a wireless communication device, that includes, but notis limited to, an integrated circuit (IC), chip set, chip,system-on-a-chip including low noise amplifier, power amplifier,Application Specific Integrated Circuit (ASIC), digital integratedcircuits, a transceiver, receiver, or transmitter, dynamic, static ornon-transitory memory device(s), one or more computer processor(s) toprocess received and transmitted signals, for example, to and from theInternet, other wireless devices, and to provide communication withinthe wireless information communication and delivery system includingsend, broadcast, and receive information, signal data, location data, abus line, an antenna to transmit and receive signals, and power supplysuch as a rechargeable battery or power storage unit. The chip or IC maybe constructed (“fabricated”) on a “die” cut from, for example, aSilicon, Sapphire, Indium Phosphide, or Gallium Arsenide wafer. The ICmay be, for example, analogue or digital on a chip or hybrid combinationthereof. Furthermore, digital integrated circuits may contain anythingfrom one to thousands or millions of signal invertors, and logic gates,e.g., “and”, “or”, “nand” and “nor gates”, flipflops, multiplexors,etc., on a square area that occupies only a few millimeters. The smallsize of, for instance, IC's allows these circuits to provide high speedoperation, low power dissipation, and reduced manufacturing costcompared with more complicated board-level integration.

As used herein, the terms “wireless”, “wireless data transfer,”“wireless tracking and location system,” “positioning system” and“wireless positioning system” refer without limitation to any wirelesssystem that transfers data or communicates or broadcasts a message,which communication may include location coordinates or otherinformation using one or more devices, e.g., wireless communicationdevices.

As used herein, the terms “module” or “modules” refer without limitationto any software, software program(s), firmware, or actual hardware orcombination thereof that has been added on, downloaded, updated,transferred or originally part of a larger computation or transceiversystem that assists in or provides computational ability including, butnot limited to, logic functionality to assist in or providecommunication broadcasts of commands or messages, which communicationmay include location coordinates or communications between, among, or toone or more devices, e.g., wireless communication devices.

FIG. 1 is a flow diagram showing a method 100 of visual odometry, inaccordance with an embodiment.

In some embodiments in accordance with the present disclosure, anon-transitory, i.e., non-volatile, computer readable storage medium isprovided. The non-transitory computer readable storage medium is storedwith one or more programs. When the program is executed by theprocessing unit of a computing device, i.e., that are part of a vehicle,the computing device is caused to conduct specific operations set forthbelow in accordance with some embodiments of the present disclosure.

In some embodiments, as illustrated in FIG. 9, examples ofnon-transitory storage computer readable storage medium may includemagnetic hard discs, optical discs, floppy discs, flash memories, orforms of electrically programmable memories (EPROM) or electricallyerasable and programmable (EEPROM) memories. In certain embodiments, theterm “non-transitory” may indicate that the storage medium is notembodied in a carrier wave or a propagated signal. In some embodiments,a non-transitory storage medium may store data that can, over time,change (e.g., in RAM or cache).

In some embodiments in accordance with the present disclosure, inoperation, a client application is transmitted to the computing deviceupon a request of a user, for example, by a smart phone 910 (see FIG.9). For example, the first client device 910 may be a smart phonedownloading the application from a computer server. In operation, theapplication is installed at the vehicle. Accordingly, specific functionsmay be executed by the user through a computing device, such ascalibrating sensors and time synchronization, and, for example, sendingand receiving calibration files for data alignment purposes.

In particular, referring to FIG. 1, in operation 102, data alignment,which includes sensor calibration and time synchronization, isperformed. A vehicle is equipped with multiple complementary sensorswhich require calibration in order to represent sensed information in acommon coordinate system. In an embodiment, sensors employed in themethod include a light detection and ranging (LiDAR) sensor, one or morecameras such as monocular cameras or stereo cameras, and an inertialnavigation module. The LiDAR sensor and the cameras are mounted on theroof of the vehicle. LiDAR sensors have become increasingly common inboth industrial and robotic applications. LIDAR sensors are particularlydesirable for their direct distance measurements and high accuracy. Inan embodiment according to the present disclosure, the LIDAR sensor isequipped with many simultaneous rotating beams at varying angles, forexample, a 64-beam rotating LiDAR. The multiple-beam LiDAR provides atleast an order of magnitude more data than a single-beam LiDAR andenables new applications in mapping, object detection and recognition,scene understanding, and simultaneous localization and mapping (SLAM).

The inertial navigation module in an embodiment according to the presentdisclosure includes a global navigation satellite system (GNSS)-inertialmeasurement unit (IMU) module or an IMU-global positioning system (GPS)module. The GNSS satellite signals are used to correct or calibrate asolution from the IMU. The benefits of using GNSS with an IMU are thatthe IMU may be calibrated by the GNSS signals and that the IMU canprovide position and angle updates at a quicker rate than GNSS. For highdynamic vehicles, IMU fills in the gaps between GNSS positions.Additionally, GNSS may lose its signal and the IMU can continue tocompute the position and angle during the period of lost GNSS signal.The two systems are complementary and are often employed together. Anintegrated navigation system consisting of IMU and GPS is usuallypreferred due to the reduced dependency on GPS-only navigator in an areaprone to poor signal reception or affected by multipath. The performanceof the integrated system largely depends upon the quality of the IMU andthe integration methodology. Considering the restricted use of highgrade IMU and their associated price, low-cost IMUs are becoming thepreferred choice for civilian navigation purposes. MEMS based inertialsensors have made possible the development of civilian land vehiclenavigation as it offers small size and low-cost.

The data alignment among the sensors includes calibrating intrinsicparameters of the camera, and calibrating extrinsic parameters betweenthe camera and the inertial navigation module. Moreover, transformationbetween the inertial navigation module and LiDAR coordinate may beachieved by a method similar to that described in “UnsupervisedCalibration for Multi-beam Lasers” by Levinson, Jesse and SebastianThrun, Experimental Robotics, Springer Berlin Heidelberg, 2014.Modifications made in the method 100 include, for example, the intrinsicparameters of each beam are calibrated in advance using a supervisedmethod. Also, LiDAR scans are collected in the form of sweep. A sweep isdefined as a scan coverage of the LiDAR sensor rotating from 0 degree to360 degrees. Moreover, motion distortion within the sweep is correctedassuming that the angular and linear velocity of the LiDAR motion isconstant. In an embodiment, means for alleviating motion distortion isemployed for points of LiDAR to ensure every point in the same sweep ofLiDAR has an identical timestamp.

It is assumed that the environment is generally static and contains some3D features, i.e., it is not just smooth ground. In order to achieve anaccurate calibration, LiDAR measurements are recorded as the vehicletransitions through a series of known poses. Global pose information isirrelevant, as there is no existing map, so only local pose informationis required. Local pose data may be acquired in any number of ways, e.g.from a wheel encoder and IMU, from an integrated GPS/IMU system, or froma GPS system with real-time corrections.

Furthermore, transformation between the cameras and the LiDAR coordinatemay be calibrated using a method similar to that described in “AutomaticCamera and Range Sensor Calibration Using a Single Shot” by Geiger,Andreas, et al., Robotics and Automation (ICRA), 2012 IEEE InternationalConference on. IEEE, 2012. Modifications made in the method 100 include,for example, the intrinsic parameters of the cameras are calibrated inadvance using a method described in “A Flexible New Technique for CameraCalibration” by Z. Zhang, IEEE Transactions on Pattern Analysis andMachine Intelligence, 22(11):1330-1334, 2000. Also, the cameras includemonocular cameras, which are calibrated by multiple shots instead ofsingle shot. Moreover, registration is made by minimizing reprojectionerror and translation norm.

In addition to the calibration and transformation, time synchronizationamong the LiDAR sensor, cameras and inertial navigation module isachieved. Specifically, time synchronization between the LiDAR sensorand the inertial navigation module, between the inertial navigationmodule and the cameras, and between the LiDAR sensor and the cameras isachieved. In an embodiment, a time trigger is used to synchronize theLiDAR and cameras to ensure data alignment.

After data alignment is performed, in operation 104, these sensors areused to collect data in an environment. In an embodiment, images of theenvironment are captured by the cameras in approximately 30 Hz. LiDARscans are collected in the form of a sweep in approximately 20 Hz.Vehicle poses, including position and orientation, are collected in an“east north up” (ENU) coordinate by the inertial navigation module inapproximately 50 Hz.

In operation 106, based on the data from the sensors, machine learningis performed in a visual odometry model. Inputs to the visual odometrymodel for machine learning include images obtained by the cameras andpoint clouds obtained by the LiDAR. In an embodiment, for monocularcameras, consecutive RGB image frames in pairs are input. In anotherembodiment, for stereo cameras, RGB images with depth information(RGB-D) are input. In machine learning, convolutional neural networks(CNNs) have become popular in many fields of computer vision. CNN hasbeen widely applied to classification, and recently presentedarchitectures also allow for per-pixel predictions like semanticsegmentation or depth estimation from single images. In an embodiment,as will be further discussed, a method of training CNNs end-to-end tolearn predicting an optical flow field from a pair of images isdisclosed.

In operation 108, a prediction of static optical flow for a pair ofinput image frames is generated. Moreover, in operation 110, a set ofmotion parameters for estimating a motion between the pair of inputimage frames is generated.

Subsequently, in operation 112, the visual odometry model is trained byusing at least one of the prediction of static optical flow and themotion parameters.

FIG. 2 is a block diagram of a system 200 for visual odometry, inaccordance with an embodiment.

Referring to FIG. 2, the system 200 includes a visual odometry model 24.The visual odometry model 24 includes one or more neural networks 241and one or more merge modules 242. The neural networks may furtherinclude convolution neural networks (CNNs), deconvolution neuralnetworks (DNNs) and a recurrent neural network (RNN). Moreover, themerge modules may include merge layers in the neural networks. Inoperation, the visual odometry model 24 receives images 201 from acamera and point clouds 202 from a LiDAR. Given vehicle poses 203 froman IMU-GPS module, the images 201 and the point clouds 202 are trainedin the visual odometry model 24. The images 201, input in pair to thevisual odometry model 24, are matched against the point clouds 202. Thevisual odometry model 24, in response to the images 201 and point clouds202, generates a prediction of static optical flow 207 and a set ofmotion parameters 208, which in turn may be used to train the visualodometry model 24.

FIG. 3A is a block diagram showing the system 200 illustrated in FIG. 2in more detail.

Referring to FIG. 3A, the visual odometry model 24 in the system 200includes a first neural network 31, a second neural network 35 and athird neural network 38. The first neural network 31 further includes afirst CNN 311, a second CNN 312, and a first merge module 310 betweenthe first CNN 311 and the second CNN 312. The first CNN 311 isconfigured to, in response to a pair of consecutive image frames 201,extract representative features from the pair of consecutive imageframes 201. The first merge module 310 is configured to merge therepresentative features. In an embodiment, the representative featuresare merged by a patch-wise correlation, which is similar to thatdescribed in “Flownet: Learning Optical Flow with ConvolutionalNetworks” by Fischer et. al., arXiv preprint arXiv:1504.06852(hereinafter referred to as “the reference”). In another embodiment, therepresentative features are merged by a simple concatenation. The secondCNN 312 then decreases the feature map size. An output of the second CNN312 constitutes a portion of a motion estimate 206.

The second neural network 35 further includes a first DNN 351, a secondDNN 352 and a second merge module 350. The first DNN 351 is configuredto, in response to an output from the second CNN 312, generate a firstflow output for each layer at a first resolution. The first flow output,which may have a relatively low resolution, constitutes another portionof the motion estimate 206. The second merge module 350 is configuredto, in response to the output from the second CNN 312 and the first flowoutput from the first DNN 351, merge these outputs by, for example, apatch-wise correlation or alternatively a simple concatenation aspreviously discussed, resulting in the motion estimate 206. The secondDNN 352 is configured to, in response to the first flow output from thefirst DNN 351, generate a first flow output for each layer at a secondflow output for each layer at a second resolution. The second resolutionis higher than the first resolution. The second flow output, which mayhave a relatively high resolution, serves as a static scene opticalflow.

The third neural network 38 includes an RNN. RNN refers to a generaltype of neural network where the layers operate not only on the inputdata but also on delayed versions of hidden layers and/or output. Inthis manner, RNN has an internal state which it can use as a “memory” tokeep track of past inputs and its corresponding decisions. In anembodiment, the third neural network 38 includes a Long Short-TermMemory (LSTM) architecture. The LSTM architecture is employed to allowRNN to learn longer-term trends. This is accomplished through theinclusion of gating cells which allow the neural network to selectivelystore and “forget” memories. The third neural network 38 is configuredto, in response to the motion estimate 206 from the second merge module350 and a set of motion parameters associated with an immediatelyprevious pair of consecutive image frames (shown in FIG. 3B), generate aset of motion parameters 208 for the current pair of image frames 201.

FIG. 3B is a schematic block diagram showing operation of the system 200illustrated in FIG. 3A.

Referring to FIG. 3B, the first CNN 311 receives a first image 211 of afirst pair of consecutive image frames 201 at time t1, and extractsrepresentative features from the first image 211 of the first pair 201.Subsequently, the first CNN 311 receives a second image 212 of the firstpair 201 at time t2, and extracts representative features from thesecond image 212 of the first pair 201. The extracted representativefeatures are merged by the first merge module 310 and the mergedfeatures are reduced in feature map size by the second CNN 312. Next,the first DNN 351 generates a low-resolution flow output based on theoutput of the second CNN 312. The second merge module 350 generates afirst motion estimate 261 by merging the output of the second CNN 312and the low-resolution flow output of the first DNN 351. The second DNN352 generates a first static scene optical flow 271 based on thelow-resolution flow output of the first DNN 351. The RNN 38 generates afirst set of motion parameters 281 based on the first motion estimate261 and a set of motion parameters associated with an immediatelyprevious pair of consecutive image frames.

Similarly, the first CNN 311 receives a first image 251 of a second pairof consecutive image frames at the time t2, and extracts representativefeatures from the first image 251 of the second pair. Subsequently, thefirst CNN 311 receives a second image 252 of the second pair at time t3,and extracts representative features from the second image 252 of thesecond pair. The extracted representative features are merged by thefirst merge module 310 and the merged features are reduced in featuremap size by the second CNN 312. Next, the first DNN 351 generates alow-resolution flow output based on the output of the second CNN 312.The second merge module 350 generates a second motion estimate 262 bymerging the output of the second CNN 312 and the low-resolution flowoutput of the first DNN 351. The second DNN 352 generates a secondstatic scene optical flow 272 based on the low-resolution flow output ofthe first DNN 351. The RNN 38 generates a second set of motionparameters 282 based on the second motion estimate 262 and a set ofmotion parameters 281 associated with the first pair of consecutiveimage frames 201.

In some existing approaches, hand-crafted features are employed toextract keypoints and descriptors, and find matching points to solvemotion parameters. Such feature-based approaches may fail when the scenehas no salient keypoints. In the present disclosure, an end-to-endtrained deep network is employed for estimating motion parameters.Sequence learning sub-network can eliminate accumulated errors. Flow isused to enhance motion estimation because motion has a strong connectionwith flow. The method according to the present disclosure allows thevisual odometry model to have higher generalization ability. Since thewhole network is end-to-end trained, no hand-crafted features arerequired. The network suits well for new scenarios or scenes, whilefeature-based methods fail in new scenarios and redesign of featurescosts a lot of efforts and time. As a result, the deep network is wellsuited to large amount of data because the model capacity and totallylearned model can handle big data well. Since the price of GPS devicesgoes down, GPS signals can be added to the deep network. Also, othersignals can be added into the model. The present disclosure proposes aflow prediction for predicting motion, and employs flow as additionalinformation to enhance motion estimation. In addition, the designedstructure can easily fuse additional signals, such as GPS signals.

FIG. 4 is a flow diagram showing a method 400 for visual odometry, inaccordance with still another embodiment.

Referring to FIG. 4, in operation 402, data alignment among sensorsincluding a LiDAR, cameras and an inertia navigation module such as anIMU-GPS module is performed.

In operation 404, image data are obtained from the camera and pointclouds are obtained from the LiDAR.

In operation 406, in the IMU-GPS module, a pair of consecutive images inthe image data is processed to recognize pixels corresponding to a samepoint in the point clouds.

Subsequently, in operation 408, an optical flow for visual odometry isestablished.

In this way, with the aid of LiDAR data whose accuracy improves alearning process by increasing the IMU-GPS module's ability toaccurately spot pixels in a pair of consecutive image frames, theIMU-GPS module is learning to establish an optical flow every time itprocesses image data and generates a more precise optical flow.Consequently, with sufficient training, the IMU-GPS module is able togenerate precise optical flows reflecting movement of a vehicle.

FIG. 5 is a flow diagram showing a method 500 for visual odometry, inaccordance with yet another embodiment.

Referring to FIG. 5, in operation 502, representative features from apair input images are extracted in a first convolution neural network(CNN).

In operation 504, outputs from the first CNN are merged in a first mergemodule. The outputs include the representative features of a first imageof the pair and the representative features of a second image of thepair. Moreover, the merge may be achieved by a patch-wise correlation ora simple concatenation.

Next, in operation 506, the merged features are reduced in feature mapsize in a second CNN. The output of the second CNN constitutes a portionof a motion estimate.

In operation 508, a first flow output for each layer is generated in afirst deconvolution neural network (DNN) at a first resolution. In anembodiment, the first flow output has a relatively low resolution. Thefirst flow output constitutes another portion of the motion estimate.

In operation 510, outputs from the second CNN and the first DNN aremerged in a second merge module, resulting in a motion estimate.

In operation 512, a second flow output for each layer is generated in asecond DNN at a second resolution higher than the first resolution. Inan embodiment, the second flow output has a relatively high resolutionand serves as a static scene optical flow.

In operation 514, accumulated errors are reduced in a recurrent neuralnetwork (RNN). The RNN, by sequence learning and prediction, generates aset of motion parameters for estimating motion between the pair ofconsecutive input images.

FIG. 6 is a flow diagram showing a method 600 of visual odometry, inaccordance with yet still another embodiment.

Referring to FIG. 6, in operation 602, in response to a first image of apair of consecutive image frames, representative features are extractedfrom the first image of the pair in a first convolution neural network(CNN) in a visual odometry model.

Next, in operation 604, in response to a second image of the pair,representative features are extracted from the second image of the pairin the first CNN.

In operation 606, outputs from the first CNN are merged in a first mergemodule.

In operation 608, merged features are reduced in feature map size in asecond CNN.

In operation 610, a first flow output for each layer is generated in afirst deconvolution neural network (DNN).

In operation 612, outputs from the second CNN and the first DNN are thenmerged in a second merge module.

Next, in operation 614, a second flow output for each layer is generatedin a second DNN. The second flow output serves as an optical flowprediction.

In operation 616, a set of motion parameters associated with the firstpair is generated in a recurrent neural network (RNN) in response to themotion estimate from the second merge module and a set of motionparameters associated with an immediately previous pair of input images.

In operation 618, the visual odometry model is trained by using at leastone of the optical flow prediction and the set of motion parameters.

In operation 620, it is determined if the visual odometry model issufficiently trained. If affirmative, in operation 622, the trainedvisual odometry model may enter a test mode. If not, then in operation624, another pair of consecutive image frames is received. Moreover, inoperation 626, the first set of motion parameters is provided to theRNN.

FIG. 7 is a flow diagram showing a method 700 of visual odometry, inaccordance with a further embodiment.

Referring to FIG. 7, in operation 702, in response to images in pairs, aprediction of static scene optical flow for each pair of the images isgenerated in a visual odometry model through deep learning.

In operation 704, a set of motion parameters for each pair of the imagesis generated in the visual odometry model.

In operation 706, the visual odometry model is trained by using theprediction of static scene optical flow and the motion parameters.

In operation 708, motion between a pair of consecutive image frames ispredicted by the trained visual odometry model.

FIGS. 8A and 8B are flow diagrams showing a method 800 of visualodometry, in accordance with a still further embodiment.

Referring to FIG. 8A, in operation 802, a first image of a first pair ofimage frames is received, and representative features are extracted fromthe first image of the first pair in a first convolution neural network(CNN).

In operation 804, a second image of the first pair is received, andrepresentative features are extracted from the second image of the firstpair in the first CNN.

In operation 806, outputs from the first CNN are merged in the firstmerge module.

In operation 808, the merged features are decreased in feature map sizein a second CNN.

In operation 810, a first flow output for each layer is generated in afirst deconvolution neural network (DNN).

In operation 812, outputs from the second CNN and the first DNN aremerged in a second merge module, resulting in a first motion estimate.

In operation 814, a second flow output for each layer is generated in asecond DNN. The second flow output serves as a first optical flowprediction.

In operation 816, in response to the first motion estimate, a first setof motion parameters associated with the first pair is generated in arecurrent neural network (RNN).

Subsequently, in operation 818, the visual odometry model is trained byusing at least one of the first optical flow prediction and the firstset of motion parameters.

Referring to FIG. 8B, in operation 822, a first image of a second pairof image frames is received, and representative features are extractedfrom the first image of the second pair in the first CNN.

In operation 824, a second image of the second pair is received, andrepresentative features are extracted from the second image of thesecond pair in the first CNN.

In operation 826, outputs from the first CNN are merged in the firstmerge module.

In operation 828, the merged features are decreased in feature map sizein the second CNN.

In operation 830, a first flow output for each layer is generated in thefirst DNN.

In operation 832, outputs from the second CNN and the first DNN aremerged in the second merge module, resulting in a second motionestimate.

In operation 834, a second flow output for each layer is generated inthe second DNN. The second flow output serves as a second optical flowprediction.

In operation 836, in response to the second motion estimate and thefirst set of motion parameters, a second set of motion parametersassociated with the second pair is generated in the RNN.

In operation 838, the visual odometry model is trained by using at leastone of the second optical flow prediction and the second set of motionparameters.

FIG. 9 is a block diagram of a system 900 for generating a ground truthdataset for motion planning, in accordance with some embodiments.

Referring to FIG. 9, the system 900 includes a processor 901, ancomputer server 902, a network interface 903, an input and output (I/O)device 905, a storage device 907, a memory 909, and a bus or network908. The bus 908 couples the network interface 903, the I/O device 905,the storage device 907 and the memory 909 to the processor 901.

Accordingly, the processor 901 is configured to enable the computerserver 902, e.g., Internet server, to perform specific operationsdisclosed herein. It is to be noted that the operations and techniquesdescribed herein may be implemented, at least in part, in hardware,software, firmware, or any combination thereof. For example, variousaspects of the described embodiments, e.g., the processor 901, thecomputer server 902, or the like, may be implemented within one or moreprocessing units, including one or more microprocessing units, digitalsignal processing units (DSPs), application specific integrated circuits(ASICs), field programmable gate arrays (FPGAs), or any other equivalentintegrated or discrete logic circuitry, as well as any combinations ofsuch components.

The term “processing unit” or “processing circuitry” may generally referto any of the foregoing logic circuitry, alone or in combination withother logic circuitry, or any other equivalent circuitry. A control unitincluding hardware may also perform one or more of the techniques of thepresent disclosure.

In some embodiments in accordance with the present disclosure, thecomputer server 902 is configured to utilize the I/O port 905communicate with external devices via a network 908, such as a wirelessnetwork. In certain embodiments, the I/O port 905 is a network interfacecomponent, such as an Ethernet card, an optical transceiver, a radiofrequency transceiver, or any other type of device that can send andreceive data from the Internet. Examples of network interfaces mayinclude Bluetooth®, 3G and WiFi® radios in mobile computing devices aswell as USB. Examples of wireless networks may include WiFi®,Bluetooth®, and 3G. In some embodiments, the internet server 902 isconfigured to utilize the I/O port 905 to wirelessly communicate with aclient device 910, such as a mobile phone, a tablet PC, a portablelaptop or any other computing device with internet connectivity.Accordingly, electrical signals are transmitted between the computerserver 900 and the client device 910.

In some embodiments in accordance with the present disclosure, thecomputer server 902 is a virtual server capable of performing anyfunction a regular server has. In certain embodiments, the computerserver 900 is another client device of the system 900. In other words,there may not be a centralized host for the system 900, and the clientdevices 910 in the system are configured to communicate with each otherdirectly. In certain embodiments, such client devices 910 communicatewith each other on a peer-to-peer (P2P) basis.

The processor 901 is configured to execute program instructions thatinclude a tool module configured to perform a method as described andillustrated with reference to FIGS. 1, 4 through 7, 8A and 8B.Accordingly, the tool module is configured to execute the operationsincluding: performing data alignment among sensors including a LiDAR,cameras and an IMU-GPS module; collecting image data and generatingpoint clouds; processing, in the IMU-GPS module, a pair of consecutiveimages in the image data to recognize pixels corresponding to a samepoint in the point clouds; and establishing an optical flow for visualodometry.

The network interface 903 is configured to access program instructionsand data accessed by the program instructions stored remotely through anetwork (not shown).

The I/O device 905 includes an input device and an output deviceconfigured for enabling user interaction with the system 900. In someembodiments, the input device comprises, for example, a keyboard, amouse, and other devices. Moreover, the output device comprises, forexample, a display, a printer, and other devices.

The storage device 907 is configured for storing program instructionsand data accessed by the program instructions. In some embodiments, thestorage device 907 comprises, for example, a magnetic disk and anoptical disk.

The memory 909 is configured to store program instructions to beexecuted by the processor 901 and data accessed by the programinstructions. In some embodiments, the memory 909 comprises a randomaccess memory (RAM) and/or some other volatile storage device and/orread only memory (ROM) and/or some other non-volatile storage deviceincluding other programmable read only memory (PROM), erasableprogrammable read only memory (EPROM), electronically erasableprogrammable read only memory (EEPROM), flash memory, a hard disk, asolid state drive (SSD), a compact disc ROM (CD-ROM), a floppy disk, acassette, magnetic media, optical media, or other computer readablemedia. In certain embodiments, the memory 909 is incorporated into theprocessor 901.

Thus, specific embodiments and applications have been disclosed. Itshould be apparent, however, to those skilled in the art that many moremodifications besides those already described are possible withoutdeparting from the disclosed concepts herein. The embodiment, therefore,is not to be restricted except in the spirit of the appended claims.Moreover, in interpreting both the specification and the claims, allterms should be interpreted in the broadest possible manner consistentwith the context. In particular, the terms “comprises” and “comprising”should be interpreted as referring to elements, components, or steps ina non-exclusive manner, indicating that the referenced elements,components, or steps may be present, or utilized, or combined with otherelements, components, or steps that are not expressly referenced.Insubstantial changes from the claimed subject matter as viewed by aperson with ordinary skill in the art, now known or later devised, areexpressly contemplated as being equivalent within the scope of theclaims. Therefore, obvious substitutions now or later known to one withordinary skill in the art are defined to be within the scope of thedefined elements. The claims are thus to be understood to include whatis specifically illustrated and described above, what is conceptuallyequivalent, what can be obviously substituted and also what essentiallyincorporates the essential idea of the embodiment.

What is claimed is:
 1. A method of visual odometry for a non-transitorycomputer readable storage medium storing one or more programs, the oneor more programs comprising instructions, which when executed by acomputing device, causes the computing device to perform the followingsteps comprising: performing data alignment among sensors including aLiDAR, cameras and an IMU-GPS module; collecting image data andgenerating point clouds; processing a pair of consecutive images in theimage data to recognize pixels corresponding to a same point in thepoint clouds; and establishing an optical flow for visual odometry. 2.The method according to claim 1 further comprising: receiving a firstimage of a first pair of image frames, and extracting representativefeatures from the first image of the first pair in a first convolutionneural network (CNN); and receiving a second image of the first pair,and extracting representative features from the second image of thefirst pair in the first CNN.
 3. The method according to claim 2 furthercomprising: merging, in the first merge module, outputs from the firstCNN; and decreasing feature map size in a second CNN.
 4. The methodaccording to claim 3 further comprising: generating a first flow outputfor each layer in a first deconvolution neural network (DNN).
 5. Themethod according to claim 4 further comprising: merging, in a secondmerge module, outputs from the second CNN and the first DNN, andgenerating a first motion estimate.
 6. The method according to claim 5further comprising: generating a second flow output for each layer in asecond DNN, the second flow output serves as a first optical flowprediction.
 7. The method according to claim 6 further comprising: inresponse to the first motion estimate, generating a first set of motionparameters associated with the first pair in a recurrent neural network(RNN).
 8. The method according to claim 7 further comprising: trainingthe visual odometry model by using at least one of the first opticalflow prediction and the first set of motion parameters.
 9. The methodaccording to claim 1 further comprising: receiving a first image of asecond pair of image frames, and extracting representative features fromthe first image of the second pair in the first CNN; and receiving asecond image of the second pair, and extracting representative featuresfrom the second image of the second pair in the first CNN.
 10. Themethod according to claim 9 further comprising: merging, in the firstmerge module, outputs from the first CNN; and decreasing feature mapsize in the second CNN.
 11. The method according to claim 10 furthercomprising: generating a first flow output for each layer in the firstDNN.
 12. The method according to claim 11 further comprising: merging,in the second merge module, outputs from the second CNN and the firstDNN, and generating a second motion estimate.
 13. The method accordingto claim 12 further comprising: generating a second flow output for eachlayer in the second DNN, the second flow output serves as a secondoptical flow prediction.
 14. The method according to claim 13 furthercomprising: in response to the second motion estimate and the first setof motion parameters, generating a second set of motion parametersassociated with the second pair in the RNN.
 15. The method according toclaim 14 further comprising: training the visual odometry model by usingat least one of the second optical flow prediction and the second set ofmotion parameters.